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1.0  INTRODUCTION 

The  findings  of  Sonex  Enterprises  Incorporated  under  the  Small  Business 
Innovation  Research  (SBIR)  Phase  I  program  are  contained  within  this  report. 
The  research  was  sponsored  by  the  US  Army,  Communications-Electronics 
Command,  Product  Assurance  and  Test  Directorate,  Fort  Monmouth,  New  Jersey. 

1.1  Problem  Statement 

The  problem  statement  for  this  research  was  simplified  to:  Is  it 
feasible  today  to  produce  a  sound  automated  software  test  plan  to  support 
DoD  system  development  and  test  activities?  If  it  is  feasible,  how  would  the 
testing  architecture  be  structured? 

1.2  Scope  of  the  Study 

The  current  procurement/productivity  environment  demands  a  reduction  in 
the  system  life  cycle  cost.  The  promise  of  fourth  generation  languages, 
relational  data  base  technology,  object  oriented  programming,  re-useable 
software  and  other  factors  have  led  to  systems  with  expensive  and  extended 
life  cycles.  The  procurement  environment  requires  that  the  potential  run¬ 
away  life  cycle  costs  be  constrained  by  establishing  control  at  system 
inception.  Testing  is  the  only  common  control  available  for  manager, 
programmer,  contractor  and  customer.  As  the  life  cycle  of  automated  system 
continues  to  be  extended,  the  ability  to  test  systems  at  all  phases  of 
development  is  accentuated.  At  Figure  1  is  the  graphic  depicting  the 
traditional  Effort  Distribution  for  Large  Projects  (Putnam,  1980).  We  have 
modified  this  figure  to  show  the  need  for  an  increase  and  earlier  start  of 
the  testing  effort  within  the  system  life  cycle.  Our  hypothesis  is  that 
additional  testing  must  be  included  in  the  system  definition  and  functional 
design  specification  phases  to  test  the  requirements  before  any  code  is 
written.  Testing  starts  when  the  system  definition  starts  --  because  quality 
can  not  be  incorporated  into  the  system  at  the  test  phase  near  end  of 
development,  quality  must  be  built  into  the  system.  The  emphasis  of  this 
research  has  been  to  verify  that  true  testability  could  be  included  in  the 
system  definition  phase  and  functional  design  phases  of  system  development. 


Figure  2  depicts  the  DoD  defined  levels  of  testing  and  the  requirements 
documents/ specif ications  consistant  with  MIL-STD-490  series  documentation 
definitions.  The  focus  of  this  research  is  the  Acceptance  level  testing  of 
the  B5  specification. 

1.3  Organization  of  this  Document 

Section  1  -  is  the  introduction  to  this  final  report. 

Section  2  -  provides  information  on  other  current  public  domain  research 
and  development  in  this  area. 

Section  3  -  is  a  short  discussion  of  the  study  approach  in  terms  of 
reductions  in  systems  life  cycle  cost  and  testing  as  a  control  function. 
Section  4  -  presents  the  finding  of  this  research. 

Section  5  -  is  the  conclusions 

Section  6  -  provides  the  recommendations. 

1.4  Testing  References 

DoD-STD-2167  -  Defense  System  Software  Development 
DoD-STD-2168  -  (Draft)  Software  Quality  Evaluation 

TB  18-104  -  Technical  Bulletin,  Army  Automation  Testing  of  Computer 

Software  Systems 

Mil-STD-490  -  Specification  Practices 

DoD-STD-7935  -  Automated  Data  Systems  Documentation  Standards 
ANSI/MIL-STD-1815A  -  Ada  Reference  Manual 

AMC-P  70-13  -  Army  Materiel  Command,  Software  Management  Indicators 
AMC-P  70-14  -  Army  Materiel  Command,  Software  Quality  Indicators 

2.0  OTHER  CURRENT  PUBLIC  DOMAIN  RESEARCH  AND  DEVELOPMENT 

This  section  focuses  on  representative  efforts  within  the  stated  study 
scope  of  reducing  system  life  cycle  costs  by  using  testing  as  a  control 
feature.  This  is  a  succinct,  non-exhaustive  discussion  of  current  R&D 
efforts  in  the  software  testing  metrics,  programming  environment  language 
domain,  and  the  expert,  or  knowledge  based  system  domains. 
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2.1  Software  Testing  Metrics 


Within  of  the  current  state  of  the  art  in  software  testing  metrics  are 
two  representative  metrics;  McCabes  Cyclomatic  Complexity  Metric;  and 
Halsteads  Information  Volume  Metric.  Software  metrics  are  management  tools 
scientific/ empirically  based,  that  must  be  unambiguous  and  objective  to  be 
useful.  However,  metrics  are  often  misunderstood  and  misapplied. 


McCabes  Cyclomatic  Complexity 


o  An  early  attempt  to  apply  the  notion  of  complexity  to  measure 
software  quality. 

o  Easy  to  compute  V(G)  =  Sum  (Loops,  conditions,  cases)  +  1. 
o  Language  dependent. 

o  Very  weak  when  comparing  software  of  same/similar  cyclomatic 
complexities . 

Halsteads  Information  Volume 


o  Derived  from  common  sense,  information  theory,  and  psychology 
o  Needs  automated  computation: 

V  =  N  Log2  n,  where: 

N  =  Total  Operators  and  Operands, 
n  =  Total  Unique  Operators  and  Operands, 
o  Works  for  any  algorithm  in  any  language. 

o  The  major  weakness  is  treating  user  function  calls  and  system 

function  calls  equally. 


Other  metrics  have  been  successful  in  specific  domains,  such  as  function 
point  analysis.  Of  course,  the  old  standby  metric,  is  delivered  source 
instructions  per  person  month  (lines  of  code). 


2.2  New  Environments /Languages 
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TAME  (Tailoring  an  Ada  Measurement  Environment)  is  research  ongoing  at 
the  University  of  Maryland.  This  research  aims  at  the  development  of  a 
prototype  measurement  and  evaluation  environment  that  supports  the  measure¬ 
ment  and  evaluation  of  the  quality,  productivity,  and  product  aspects  of  Ada 
projects.  TAME  includes  the  processes  of  setting  up  measurement  and 
evaluation  goals  and  deriving  supportive  measures.  The  current  prototype 
does  not  interface  with  an  (APSE)  Ada  Programming  Support  Environment; 
however,  it  is  designed  to  be  integrated  into  an  APSE  in  the  future.  The 
TAME  system  provides  means  for  collecting,  storing,  and  validating  data, 
computing  measures,  and  interpreting  computed  values  within  the  context  of 
particular  evaluation  goals. 
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(MOTHRA)  Mutation  Testing  Architecture,  is  a  prototype  environment  based 
on  the  program  mutation  testing  technology  at  Georgia  Tech.  The  environment 
is  an  integrated  set  of  tools  and  interfaces  that  support  the  planning, 
definition,  preparation,  execution,  analysis  and  evaluation  of  tests  of 
software  systems.  The  MOTHRA  environment  was  designed  with  two  primary 
objectives.  The  first  is  that  the  environment  possess  high  band  width  user 
interfaces.  The  second  is  that  it  impose  no  a  priori  constraints  on  the  size 
of  software  that  can  be  tested  in  the  environment.  In  addition,  it  supports 
the  notion  of  progressive  tests,  in  that  it  allows  the  user  to  carry  data 
from  one  level  of  testing  to  higher  levels,  with  the  capability  to  incor¬ 
porate  that  test  data  into  the  overall  test  objectives. 

Mutation  analysis  and  thus  the  MOTHRA  environment  allows  the  tester  to 
create  test  data,  evaluate  old  test  data,  detect  the  absence  of  known  errors, 
and  provides  error  detection.  These  capabilities  are  also  very  applicable  to 
the  testing  of  reusable  software.  Evaluation  of  old  data  and  the  creation  of 
new  test  data  allows  the  user  of  a  reusable  component  to  develop  test  cases 
that  are  related  to  the  operational  objectives  of  the  system,  while 
eliminating  the  generation  of  redundant  test  data  from  previous  testing. 


^  CSRL  (Conceptual  Structures  Representation  Language)  has  been  used  by  the 

Battelle  Columbus  Division  for  several  industrial  applications  as  part  of 

ft 

O'  Battelle’ s  contract  research  business  over  the  last  three  years. 

K- 


V 


6 


to: 


CSRL,  developed  by  the  Laboratory  for  Artificial  Intelligence  Research  at 
the  Ohio  State  University,  is  a  language  for  building  classification  expert 
systems.  Based  on  the  notion  that  classification  problem-solving  can  be 
modeled  as  a  society  of  specialists,  CSRL  implements  such  knowledge-based 
systems  as  a  classification  hierarchy  of  SPECIALISTS,  where  individual 
SPECIALISTS  engage  in  hypothesis  refinement,  i.e.,  the  task  of  the  problem 
solver  is  to  find  the  categories,  or  hypotheses,  within  the  classification 
hierarchy  which  is  appropriate  to  the  situation  being  analyzed.  Examples  of 
classification  problem-solving  include  diagnosis,  catalog  selection,  and 
certain  types  of  planning. 

Two  major  conclusions  have  resulted  from  using  this  language  in  an 
industrial  setting.  First,  CSRL  is  a  powerful  knowledge  engineering  tool  and 
it  also  supports  standard  software  engineering  needs  for  developing  computer 
software.  Second,  several  identified  enhancements  are  necessary  to  make  CSRL 
a  more  effective  and  cost-efficient  development  tool.  With  these 
enhancements,  CSRL  will  satisfy  the  software  engineering  goals  for  the 
development  of  expert  systems  which  are  testable,  reliable,  cost-effective, 
well-documented,  understandable,  maintainable,  and  modifiable  -  software 
asich  meets  the  user’s  needs. 

2.3  Knowledge  Based  Systems 

SAC  (Software  Acquisition  Consultant)  research  being  performed  at  the 
Naval  Underwater  Systems  Center,  New  London  laboratory,  CT .  The  goal  of  this 
research  is  to  produce  an  expert  system  decision  aid  for  tailoring  the 
requirements  of  DoD  STD  2167,  using  DoD-HDBK-287 ,  associated  Data  Item 
Descriptions  (DID),  and  standards.  The  current  prototype  is  currently 
employed  in  the  selection  of  DIDs  to  be  required  for  a  software  development 
project.  The  purpose  is  to  provide  software  acquisition  managers, 
responsible  for  applying  DoD-STD-2167 ,  with  software  engineering  expertise. 
SAC  assists  in  developing  the  appropriate  level  of  requirements  3nd 
documentation  tailored  for  each  procurement. 

DIOGENES  (Expert  System  for  Extraction  of  Data  System  Requirement s  from 
User  Scenarios).  This  NASA  SBIR  work  derives  system  requirements  from  user 


scenarios  by  facilitating  and  analyzing  interactions  between  software  systems 
engineers  and  system  end  users.  This  prototype  system  has  automated  a 
scenario-driven  methodology  for  deriving  top-level  specifications  and 
preliminary  designs  for  user  data  systems. 

Expert  System  for  Software  Quality  Assurance,  is  a  prototype  that  was 
created  for  the  US  Army  Belvoir  Research,  Development  and  Engineering 
Center.  This  expert  system  facilitates  the  process  of  tailoring  statements 
of  work  by  capturing  the  knowledge  of  software  QA  engineers.  The  system  was 
executed  to  alleviate  staff  turnover/inexperience  and  to  ensure  that  the 
consistent  standards  and  requirements  of  an  adequate  software  QA  program  are 
enforced . 

ECA  (Expert  Complexity  Analyzer)  by  Autometrics  Inc.  is  a  knowledge  based 
system  that  provides  individual  module  level  analysis,  module  clean-up 
suggestions,  bug  predictions  and  project  scheduling.  The  system  is  tailored 
to  the  DoD-STD  2167  environment  and  at  the  Preliminary  Design  Stage  will 
provide  an  initial  testing  schedule  and  an  effective  linear  based  development 
schedule  without  provision  for  software  complexity  or  defect  prediction.  At 
the  Preliminary  Design  Review  (PDR)  a  more  refined  testing  schedule, 
dynamically  based  on  software  complexity,  testing  personnel  allocation, 
initial  defect  predictions  and  resource  allocation  recommendations  is 
provided.  At  the  critical  Design  Review  (CDR)  the  system  provides  schedule 
and  testing  personnel  allocation  reflective  of  software  complexity,  full 
defect  prediction  for  tracking  test  effectiveness,  predicted  reliability 
estimates,  test  management  suggestions,  and  test  management  reports.  During 
the  Software  Testing  Phase  the  system  provides  analysis  of  defects  found  to 
those  predicted,  defects  remaining  estimates,  test  schedule  refinement,  test 
effectiveness  reports,  and  operational  availability  and  system  reliability 
predictions.  ECA  future  capabilities  will  include;  development  of  a 
historical  project  testing  database;  development  of  module,  unit  and  build 
test  strategies;  development  of  automated  test  scenarios;  integration  e: 
automated  test  path  generation  techniques. 

ASQ  (Automated  Assistant  for  Specification  of  Software  Quail  tv)  .  under 
development  at  Dynamics  Research  Corporation,  allows  acquisition  managers  to 
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cost-effectively  specify  software  quality  goals  based  only  on  their  knowledge 
of  the  application  and  system  specifications.  The  tool  provides  a  mechanism 
for  putting  technology  to  support  software  quality  specification  into  use  in 
todays ’ s  DoD  systems.  Through  automated  application  of  the  software  quality 
methodology  to  DoD  systems,  the  software  development  community  will  begin  to 
see  the  benefits  of  specifying  and  measuring  quality. 

Knowledge-based  reasoning  allows  ASQ  to  provide  the  essential  software 
quality  guidance  to  users,  and  to  incorporate  prior  decisions  made  by  the 
user  for  future  use.  The  following  are  some  of  the  important  features  of 
ASQ;  Complete,  easy-to-use  guidance,  stepping  through  each  specification 
procedure,  so  that  the  acquisition  manager  does  not  need  to  know  details  of 
the  software  quality  specification  process;  Access  to  help  for  users  of  all 
experience  levels;  A  flexible  menu-driven  user-interface;  automated 
procedures  wherever  possible;  Automated  extrapolations  from  available 
information,  whenever  a  user  skips  certain  details;  Incorporation  of 
assumptions  and  decisions  for  later  review;  Incorporation  of  a  database  of 
results  from  past  projects;  and  specification  by  example,  whenever  data  from 
example  projects  can  help  specify  quality  for  the  current  project. 

Based  upon  the  above  material  provided  on  Testing  Metrics,  new  environ¬ 
ments/languages  and  knowledge  based  developments,  we  can  see  that  there  is 
considerable  diverse  activity  in  the  software  testing  field.  However,  we 
have  not  found  any  published  reference  to  a  life  cycle  wide  testing  architec¬ 
ture  to  control  system  development  through  testing.  Our  problem  statement 
for  this  research  was;  is  it  feasible  today  to  produce  a  sound  automated 
software  test  plan  to  support  DoD  system  development  and  test  activities?  If 
it  is  feasible,  how  would  the  testing  architecture  be  structured? 


3 . 0  STUDY  APPROACH 


The  approach  for  this  research  included  five  separate  tasks;  A 
literature  review;  Expert  knowledge  execution;  Identification  of  software 
testing  requirements;  definition  of  a  software  test  case  prototype;  and 
determination  of  feasibility  of  generating  an  automated  software  test  plan. 
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3.1  Literature  Review 


The  first  task  was  to  execute  a  review  of  the  literature  and  other 
sources  to  identify  current  techniques  and  tools,  including  Knowledge  Based 
Systems  that  were  potentially  applicable  to  this  study.  The  second  task  then 
focused  on  reviewing  the  literature  to  determine  what  kind  of  progress  has 
been  achieved  in  the  automatic  software  test  plan  generation  arena.  The 
focus  of  the  effort  was  on  the  identification  of  any  knowledge  based  systems 
in  existence  that  are  applicable,  or  that  may  be  modified  to  be  applicable  to 
the  problem  area. 

Whenever  applicable  systems  did  exist,  they  were  evaluated  in  terms  of 
applicability  to  a  particular  phase  of  the  process  (e.g.,  most  probable  error 
statistics  problem)  or  to  the  overall  Testing  Architecture  process  being 
researched  under  this  contract. 

3.2  Expert  Knowledge  Extraction 

Defining  the  bounds  and  the  scope  of  the  software  test  plan  generation 
process  was  one  of  the  critical  tasks  of  this  research.  The  act  of 
extracting  the  detailed  experience  from  experts  has  a  magnifying  effect  upon 
the  typical  problem  definition.  Based  upon  previous  knowledge  based  system 
development  experience,  the  problem  statement  must  be  very  focused  because 
detailed  experience  brings  many  "new"  factors / sub-problems  to  bear  as  a  part 
of  the  ultimate  problem  solution  set. 

Figure  3  depicts  the  overall  "problem  space"  of  the  software  test  plan 
generation  problem  at  the  top  of  the  graphic.  The  problem  space  includes  the 
entire  undefined  testing  environment.  Although  based  on  Sonex’s  experience, 
in  testing  the  current  Army  Advanced  Field  Artillery  Tactical  Data  System 
(AFATDS)  development,  it  was  obvious  that  automatic  test  data  generation  was 
beyond  the  scope  of  this  research,  however,  the  issue  of  automated  test 
procedures  was  less  clear.  The  expert  interviews  progressed  in  two  stages, 
first  general  testing  issues,  which  culminated  in  the  identification  of  the 
test  plan  generation  problem  as  the  key.  The  second  stage  was  the  detailed 
definition  of  the  software  test  plan  generation  problem. 
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3.3  Software  Testing  Requirements 


After  the  key  problem  was  identified,  defined  and  approved  by  the  COTR, 
expert  interviews  continued  with  a  focus  on  the  Software  Test  Plan  to 
determine  detailed  testing  requirements.  Interview  role  relationships  were 
defined,  the  experts  were  provided  information  on  the  research  prior  to  the 
initial  meeting,  and  the  sessions  were  taped  and  transcribed.  This  interview 
process  was  complemented  with  additional  research  in  search  of  material  and 
methodologies  for  additional  points  of  view  and  academic  depth. 

3.4  Definition  of  the  Test  Case  Prototype 

The  initial  methodology  for  the  test  case  prototype  was  as  follows: 

o  Create  a  paper  system  that  allows  the  domain  expert  to  identify  and 
fill  the  voids. 

o  Determine  the  verification  criteria  for  subsequent  evaluation. 

o  Program  the  system  using  appropriate  Knowledge  Based  system 
software . 

o  Iterate  the  development  with  the  domain  expert  to  demonstrate 
various  capabilities. 

o  The  prototype  shall  solve  a  portion  of  the  vital  subset  problem 
suggesting  that  the  approach  is  viable  and  further  system 
development  is  achievable. 

However,  the  magnitude  of  the  task  (to  define  all  of  software  testing) 
coupled  with  the  requisite  thoroughness  of  the  interview  process, 
necessitated  an  alteration  to  the  research  scope.  Consequently,  the  effort 
was  refocused  on  defining  a'  testing  architecture,  rather  than  developing  a 
demonstration  prototype  knowledge  based  system. 

3.5  Expert  System  Feasibility 

The  feasibility  of  the  software  testing  architecture /Knowledge  based 
system  concept  was  developed  and  evaluated  at  the  midpoint  of  the  effort. 
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Specific  problems  that  were  expected  to  be  encountered  in  implementing  the 
system  were  identified  and  the  methodology  used  to  address  these  problems  was 
discussed.  In  addition,  the  potential  for  expanding  the  Software  Testing 
Architecture  to  include  other  aspects  of  Software  product  assurance  was 
evaluated . 

4 . 0  FINDINGS 

4.1  Literature  Review  and  State  of  the  Art  Survey 

The  research  consisted  of  deliberate  bibliographic  literature  searchs  by 
Defense  Technical  Information  Center  (DTIC)  on  AI  and  Software  Testing  (at 
Attachment  1)  and  the  Data  Analysis  Center  for  Software  (RADC/COED)  at 
Griffiss  AFB,  NY  (at  Attachment  2).  Interviews  with  industry  and  academics 
in  the  software  development  and  testing  areas  were  conducted  with  AT&T 
Federal  Systems,  IBM  Federal  Systems,  Boeing  Computer  Services,  IMR  Systems 
Corporation  and  George  Mason  University.  Sonex  has  also  remained  abreast  of 
current  technologies  through  attendance  at  the  Washington  DC  Chapter  of  ACM 
SIGAda  meetings,  and  attendance  at  representative  conferences  such  as  the 
annual  IEEE  Expert  Systems  in  Government  Symposium,  National  Conference  on 
Ada  Technology  and  Washington  Ada  Symposium,  and  National  Institute  for 
Software  Quality  and  Productivity’s  recent  Software  Testing  and  Validation 
Conference . 

From  these  activities  it  was  concluded  that  tremendous  progress  is  being 
made  in  many  diverse  fields.  However,  two  separate  forces  are  currently 
afield:  Congress  is  imposing  severe  restrictions  on  defense  spending  and  the 

spectacular  progress  in  various  fields  has  the  potential  to  create  a  run-away 
engine.  As  Ada,  and  other  new  technologies,  has  been  implemented  in  major 
software  development  projects,  the  impact  has  been  greater  than  the 
government  or  contractors  anticipated.  Ada  has  caused  both  contractor  and 
government  personnel  to  reassess  the  entire  waterfall  development  methodo¬ 
logy.  Testing  is  the  only  common  control  that  transcends  languages,  metrics, 
and  methodologies.  The  needs  for  test  control  and  discipline  have  never  been 
greater . 
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Specific  high-level  rules  of  thumb  that  emerged  from  this  literature 
review  and  state  of  the  art  survey  include: 

o  Because  of  sophistication,  resource  and  time  constraints,  software 
cannot  be  tested  exhaustively. 

o  Testing  requires  mechanization  if  it  is  to  maxe  a  serious  impact  and 
control  the  software  development  effort. 

o  The  obvious  cost  benefit  of  early  error  detection  justifies  testing 
the  written  requirements  for  logic  errors  and  determining  the 
testability  of  the  proposed  system  before  any  code  is  written. 

o  The  special  problems  of  real  time  systems  have  yet  to  be  solved. 

o  The  advantages  of  writing  the  system  users  manual  during  the 

requirements  phase,  and  its  use  as  a  testing  ground  truth. 

o  Quality  cannot  be  inserted  at  testing,  it  must  be  built  into  the 
software  product. 

o  The  phenomenon  of  defect  clustering  where  80Z  of  the  errors  are 
identified  in  20Z  of  the  code,  holds  regardless  of  the  PDL. 

From  the  level  of  actively  detected  it  is  evident  that  software  testing  is 
emerging  as  potentially  the  most  comprehensive  control  function. 

A. 2  Experts  Interviewed 

Mr.  Steven  J.  Callas  has  over  eighteen  years  industry  analysis  and 
management  experience  including  Test  Plan  development  and  implementation  for 
hardware  and  software,  customized  application  development  including  QA 
activities,  and  hands-on  experience  with  IW  and  CM. 

Increased  project  productivity  by  11Z  due  to  project  IV&V  efforts  over 
two  years.  Generated  test  plans  for  the  Army’s  independent  testing  of 
Tactical  Analysis  software  systems  and  produced  a  written  report  describing 
the  results.  Evaluated  documentation  and  provided  hardware  and  software 
configuration  management  to  the  Tactical  Analysis  System.  Reported  to  the 
Configuration  Control  Board  (CCB)  of  the  project  for  the  preparation  and 
distribution  of  CCB  decisions  and  supporting  data  requirements.  Created  a 
data  base  on  a  VAX  11/780  for  maintenance  of  configuration  management 
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information.  Prepared  configuration  management  plans  for  contract  proposals 
using  Department  of  Defense  Standards  7935  and  480;  and  Military  Standards 
481,  483,  and  490.  QA  experience  using  Army  TB  18-102  and  Navy  standards 
identifying  quality  factors  and  how  they  directly  impact  the  project  at 
hand.  Employed  QA  as  a  process,  not  as  a  checklist.  Mr.  Callas  is  currently 
advising  the  Magnavox  Corporation  on  testing  and  configuration  management  for 
the  AFATDS  System  development. 

Mr.  William  J.  Slobodian  has  over  seven  years  experience  in  the  software 
testing,  quality  assurance  and  IV&V  areas.  As  Principal  Engineer,  he  has 
written  software  test  plans  and  procedures  for  the  US  Army  Technical  Control 
and  Analysis  Center  (TCAC)  and  the  TOP  GALLANT,  SIGINT/EW  systems  of  the 
Joint  Tactical  Fusion  Program. 

Duties  included  the  performance  analysis  of  baseline  software  and 
firmware.  Analyzed  detailed  software  design  strategies  for  integrity  of 
functionality  and  system  architecture.  Reviewed  all  software  documentation 
pertaining  to  application  and  vendor  supplied  software.  Performed  manpower 
and  cost  analysis  in  relation  to  software  testing  and  other  functions. 

Served  as  engineer  specialist  for  C3I  Special  Projects.  Advised  the 
Government  Program  Manager  on  software  scheduling,  hardware  acquisitions,  and 
technical  aspects  of  the  project. 

As  Systems / Sof tware  Engineer,  he  had  duties  including  the  design,  debug 
and  implementation  of  software  for  Naval  Weapon  System  WDS  MK14.  Computer 
languages  used  on  software  project  included  CMS02M  and  Ultra-16  for  the 
Sperry  Univac  AN-UYK20  computer.  Work  included  insuring  systems  maintenance 
tests  operated  according  to  designated  specifications.  Served  as  technical 
field  representative  at  the  Land  Based  Test  Center,  Wayland,  MA.  Specific 
assignments  included  system  evaluations  on  launcher  and  fire-control 
systems.  Prepared  monthly  reports,  trained  members  of  section  on  operation 
of  OJ-194  and  USQ-69  shipboard  and  AN-UYK  20  operations. 
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Mr.  John  S.  Williams  has  a  wide  and  varied  background  of  management  of 
complex  projects  and  supervision  of  military  and  technical  organizations.  He 
has  extensive  experience  in  the  definition  and  development  of  command  and 
control  information  systems  in  a  career  marked  by  innovation  and  achievement 
of  objectives.  As  project  manager  for  defining  the  Command  and  Control 
Information  System  (C2IS)  for  Allied  Command  Europe,  he  initiated  the  use  of 
modern  structured  methodology  which  has  been  adopted  as  the  NATO  standard. 
With  direct  supervision  of  a  multi-national  team  of  systems  analysts  and 
functional  experts  tasked  to  define  strategic  C2IS  functional  and  technical 
requirements,  he  presented  and  justified  system  requirements  before  NATO’s 
Technical  and  Budget  committees  which  resulted  in  full  funding  support. 

erved  as  Chairman  of  working  groups  and  committees  responsible  for  the 
definition  of  ADP  standards,  policy  and  procedures. 

As  Chief  of  ADP  Quality  Assurance,  Supreme  Headquarters  Allied  Powers 
Europe,  Mr.  Williams  conceived,  developed  and  implemented  a  quality  assurance 
philosophy  and  procedures  which  resulted  in  marked  improvement  of  Command  am' 
Control  Information  Systems  in  operation  and  being  developed  to  support 
SHAPE.  Introduced  improved  Configuration  Management  procedures  and  tools. 

Mr.  Williami  currently  consults  on  testing  and  functional  user  issues  for 
Magnavox  Corporation,  the  AFATDS  system  development. 

Mr.  Michael  J.  Xenos  has  30  years  experience  in  IW,  CM,  and  QA 
(including  T&E)  from  which  to  draw  upon.  Mr.  Xenos  conceived  and  developed 
the  Independent / Integrated  Systems  Assurance  (ISA)  concept  which  integrates 
QA  and  CM  processes  into  a  structured  verification  and  validation 
methodology.  Pertinent  experience  includes: 

Deputy  Program  Manager  for  the  development,  operations,  and  maintenance 
of  the  US  General  Accounting  Office  (GAO)  Consolidated  Administrative 
Management  Information  System  (CAMIS),  a  $23  million  nationwide  interactive 
system.  He  directed  the  program  startup,  including  securing  and  orienting 
the  requisite  resource,  establishing  resource  accounting  policies  and 
procedures,  and  establishing  professional  relations  with  the  customer. 
Negotiated  agreements  with  subcontractors  and  managed  their  personnel 
resources.  Managed  deliverables  to  schedule  and  established  the  award  fee 
criteria. 


Project  Manager  for  development  and  implementation  of  major  enhancements 
to  the  Federal  Guaranteed  Student  Loan  (GSL)  and  the  national  Direct  Student 
Loans  (NDSL)  programs  ($23  million)  under  a  fixed-priced  contract  with  the 
Department  of  Education.  He  managed  both  company  and  subcontractor  personnel 
for  development,  implementation,  and  maintenance.  Coordinated  the 
telecommunications  network  and  hardware  requirements  with  regional  venders 
and  telephone  companies.  Represented  corporation  in  weekly  Regional 
Administrators  conferences  with  the  Commissioner  of  Student  Financial 
Assistance.  Conceived,  orchestrated  and  supervised  the  development  of  the 
Army’s  PROBE  system  that  assists  DA  in  the  automated  development  of  the 
annual  five-year  programs  and  budgets. 

Chief  Resource  Programs  (Manpower,  Equipment,  Facilities,  Dollars)  US 
Army,  Europe  (1976  -  1977).  Mr.  Xenos  managed  a  staff  of  28  with 
responsibilities  that  included:  Established  the  automated  functional 
resources  planning  structure,  established  criteria,  standards,  and  procedure'-- 
for  resource  planning,  initiated  development  of  the  European  Five-Year 
Resource  Requirements  Document,  coordinated  requirements  with  NATO 
headquarters  and  incorporated  US  commitments  to  NATO.  Mr.  Xenos  represented 
the  Command  at  Headquarters  DA  resource  allocation  boards,  integrated 
European  resource  programs  into  the  Army  Five-Year  Program  and  developed 
requirements  for  ADP  systems  to  support  resource  planning. 

Adjunct  Professor  of  Management,  Computer  Sciences  and  System  Analysis, 
for  Boston  and  American  Universities.  Developed  and  taught  a  graduate  level 
integrated  analysis  and  computer  architecture  courses  for  13  years. 

The  expertise  of  these  key  participants  was  complemented  with  short 
interviews  with  other  testing  experts.  The  result  of  these  interviews  and 
the  derived  testing  architecture  are  provided  in  the  next  section. 
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4.3  Software  Test  System  Architecture 


This  section  is  segmented  up  into  five  major  sections.  These  sections 
discuss  the  EXTEND  (Expert  Test  Evaluation  Node  Development)  system  results 
of  the  SBIR  Phase  I  research.  The  five  sections  are  supported  by  high  level 
system  architecture  charts,  the  EXTEND  Execution  Flow,  Input  and  Output 
descriptions  grouped  by  process,  a  sample  session  with  the  prototype  expert 
system  shell,  and  a  discussion  of  feasibility. 

4.3.1  EXTEND  Overview 

The  overall  system  architecture  diagram  is  at  Figure  4.  The  overall 
EXTEND  system  architecture  chart  depicts  the  main  elements  of  the  software 
testing  environment.  The  Execution  flow  of  the  EXTEND  system  will  be 
discussed  in  Section  4.3.2.  This  architecture  begins  with  a  subsystem  that 
addresses  the  potential  system  target  environment  and  then  identifies 
appropriate  Computer  Software  Configuration  Item  (CSCI)  structures.  The 
proposed  CSCI  decision  tree  design  impacts  significantly  on  software 
testability . 

The  CSCI  tree  feeds  the  Software  Test  Development  subsystem.  Other  major 
inputs  are  Software  Metric  Model  profiles,  T  Tool  (discussed  in  Section 
4. 3. 3. 3)  requirements  and  test  data  cases.  The  Test  Development  Subsystem  is 
the  focal  item  in  the  EXTEND  Architecture.  The  software  test  development 
process  is  further  defined  in  Figure  5.  The  software  test  products  of  the 
test  development  process  are  provided  to  a  generic  testbed  or  testbed 
interface  and  an  off-the-shelf  tools  interface.  The  final  product  of  the 
EXTEND  architecture  is  the  Test  Results  Analysis  Subsystem  that  analyzes  the 
tests  for  acceptance.  This  sub-system  also  provides  feedback  to  other 
components  of  the  EXTEND  architecture. 

The  detailed  Test  Development  Subsystem  diagram  at  Figure  5  is  composed 
of  three  major  functions.  The  generic  flow  of  control  begins  with  the  Test 
Resources  Subsystem  and  moves  to  the  Test  Plans  subsystem.  Based  upon  these 
inputs,  the  final  player  is  the  Test  Procedures  subsystem. 
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4.3.2  EXTEND  Execution  Flow 


The  EXTEND  System  architecture  has  the  capability  to  always  produce  a 
software  test  plan.  This  software  test  plan  shall  be  repeatable  and  one  of 
the  software  test  plan  appendices  will  be  the  Assumptions  and  Constraints 
invoked  to  create  the  software  test  plan.  The  extent  of  these  appendices  is 
a  function  of  development  phase.  The  Interactive  session  between  the  user 
and  the  EXTEND  system  will  begin  by  defining  the  environment  of  the  target 
system  (see  Figure  6).  If  there  is  information  the  EXTEND  system  requests 
and  the  user  doesn’t  know,  EXTEND  will  derive  a  default  value,  based  on 
expert  experience,  rules,  and  the  previous  session  input.  The  architecture 
of  the  EXTEND  system  requires  a  minimum  number  of  data  inputs  to  create  the 
resultant  test  plan.  If  the  user  can  not  provide  the  input  data,  then  the 
interactive  EXTEND  system  module  assigns  default  condition  values  based  upon 
Expert  tester  experience  assumptions.  The  basis  for  these  assumptions  will 
come  from  Sonex  research/interviews  with  testing  experts  and  data  bases  of 
historic  testing  experience.  The  assumptions  for  the  specific  session  shall 
be  provided  as  an  appendix  of  the  resultant  Software  test  plan. 

Likewise,  when  the  user  can  answer  specific  questions  about  the  target 
environment,  then  questions  about  the  CSCI  structure  of  the  target  system 
will  be  asked  (also  depicted  in  Figure  6).  This  interaction  about  the  CSCI 
structure  will  question  the  user  about  the  target  system  requirements  and  the 
options  for  target  system  design.  Based  upon  the  ability  of  the  user  to 
respond,  for  example,  that  the  detail  design  data  is  available,  default 
conditions  and  assumptions  will  be  made  for  missing  user  inputs.  These 
default  values  and  conditions  and  the  resultant  assumptions  they  are  based 
upon,  will  be  provided  to  the  user  as  part  of  the  software  test  plan 
Assumptions  and  Constraints  appendices. 

The  results  from  either  user  input  or  default  values  will  next  activate 
the  Metric  Model  and  T  Tool  subsystems.  The  Metric  Model  subsystem  will 
query  the  user  to  extract  the  minimum  data  inputs  that  the  Metric  Model 
requires  to  make  a  software  testing  determination.  Responses  that  the  user 
can  not  accurately  detail  will  have  default  values  associated  with  them. 
Again,  the  default  values / conditions  will  be  documented  and  the  supporting 
assumptions  provided  as  a  component  of  the  software  test  plan  Appendices  for 
Assumptions  and  Constraints. 
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EXTEND  EXECUTION  FLOW 


The  minimum  data  inputs,  required  to  create  the  software  test  plan  came 
from  the  user,  or  through  the  default  condition’s  mechanism  that  is  generated 
in  the  Target  Environment,  CSCI  Analyzer,  and  Metric  Model  Subsystems.  The 
appropriate  minimum  inputs  are  now  provided  to  the  T  Tool  to  generated 
requirements  traceability,  test  inputs,  intermediate  test  results  and 
expected  test  results.  Additional  assumptions  and  constraints  when  default 
values  are  used  will  be  generated  to  substantiate  the  T  Tool  results. 

The  result  of  the  T  Tool,  and  CSCI  analyzer,  and  the  Metric  Model  are 
next  provided  to  the  Test  Development  subsystem  to  produce  the  Software  Test 
Plan.  The  minimum  data  inputs  are  provided  by  the  user  oj:  the  EXTEND  system 
as  default  values /conditions .  These  defaults  will  be  provided  with  backup 
assumptions  that  define  the  default  rationale  and  will  be  provided  as  an 
appendix  to  the  Software  Test  Plan. 

This  architecture  will  produce  a  Software  Test  Plan  for  each  formal 
test.  The  assumptions  and  constraints  that  validate  the  output  software  test 
plan  would  also  be  provided  as  an  appendix  consistant  with  Dl-MCCR-80014 . 

The  EXTEND  System  architecture  will  allow  the  creation  of  a  software  test 
plan,  as  a  living  document,  during  any  phase  of  the  target  system 
development.  The  EXTEND  system  could  be  used  as  the  tester  progresses 
through  the  testing  process,  allowing  the  tester  to  use  the  EXTEND  results  as 
feedback  to  change  his/her  original  user  inputs.  The  changed/more  detailed 
user  input  then  allows  for  less  EXTEND  default  values / conditions  and  provide 
for  a  refocused  testing  strategy.  The  EXTEND  system  will  always  produce  the 
same  result  with  the  same  inputs.  The  effects  of  time,  and  more  available 
details  about  the  target  system,  allow  for  a  more  detailed  software  test 
plan.  This  more  detailed  software  test  plan  will  generally  have  less 
assumptions  and  constraints  attached  when  the  user  provides  more  detail. 

The  EXTEND  system  could  be  available  to  assist  in  testing  decisions,  based 
upon  modified  user  input. 

During  an  update  session,  the  user  input  about  the  target  system  would 
produce  a  new  software  test  plan,  appendix  of  assumptions  and  constraints, 
and  allow  a  paper  (hard  copy)  comparison  of  the  various  sessions.  This 
comparison  would  also  provide  the  testing  knowledge  base  explanation  and 
rationale  for  the  results. 


23 


The  EXTEND  System’s  knowledge  base  would  require  a  very  extensive 
infusion  of  experience  to  provide  a  software  test  plan.  The  mapping  of  the 
required  knowledge  and  identification  of  holes  in  the  knowledge,  will  be  aided 
by  use  of  the  EXTEND  system  itself.  The  subsystem  knowledge  bases  will 
require  updating  as  holes  and  voids  are  identified,  and  as  default  conditions 
and  assumptions  are  critiqued  by  experts  during  the  EXTEND  testing  period. 

The  development  of  the  EXTEND  architecture  would  focus  on  the  establish¬ 
ment  of  test  threads  that  allow  creation  of  the  software  test  plan.  The 
minimum  user  inputs  or  default  values  to  activate  the  test  thread  would  be 
identified  during  the  EXTEND  analysis.  Many  possible  test  threads  are 
depicted  in  Figures  4  and  5. 

4.3.3  Input/Output  and  Processes 

The  listings  described  below  provide  definition  for  the  terms  of 
processes  and  data  flows  (inputs  and  outputs)  used  in  Figures  4  and  5.  The 
research  effort  has  focused  on  defining  the  EXTEND  system  inputs/outputs  as  a 
means  of  top-down  decomposing  the  overall  EXTEND  architecture. 

4. 3. 3.1  Target  Environment 

This  process  embodies  the  current  target  system  estimate  for  the  proposed 
deployment  system,  including  size.  The  purpose  is  to  identify  design  and 
test  objectives  that  affect  CSCI  Acceptance  Testing. 

Inputs  to  the  Target  Environment  are: 

1.  High  Level  Requirements  Document:  For  example,  the  system  specification 
in  the  Military  Standard  490  series  documentation.  This  document  must 
provide  system  characteristics  and  identify  design  constraints  i_f  they 
exist,  (i.e.,  memory  size  constraints).  The  requirements  are  captured  in 
Functional  Descriptions,  Systems  Specifications  and  "A"  Specs. 
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2.  Management  Control:  The  current  testing  overview  with  resource,  time 
constraints,  and  other  factors  included.  Specific  areas  of  interest 
include  data  on  configuration  management,  milestones,  reporting,  budget, 
logistics,  support  equipment,  facilities,  personnel  and  interoperating 
systems  requirements. 

3.  System  Characteristics:  User  and  designer  current  estimates  of  the 
hardware,  software,  and  operating  system  for  the  proposed  system.  The 
selection  cf  a  DBMS,  off-the-shelf  requirements,  and  the  development 
versus  the  deployment  environment  issues  are  also  involved.  This  is 
working  information  only.  It  will  change  completely  as  the  requirements 
evolve . 

Outputs  from  the  Target  Environment  are: 

1.  Programming  Environment:  The  determination  of  a  high  order  language  cu 
assembler,  real-time  or  batch,  on-board  embedded  or  special  purpose 
environment.  (This  input  leads  to  determination  of  the  lines  of  code). 
The  complexity  of  the  control  structure  and  data  is  included  here.  A 
true  lines  of  code  projection  is  not  within  the  grasp  of  current 
technology.  However,  the  intent  is  to  size  the  software  with  a  measure. 

2.  Dominant  Quality  Factors:  The  identification  of  key  system  quality 
factors.  This  is  developed  during  a  user  and  system  interaction, 
answering  questions  on  the  dominant  system  characteristics.  Examples  of 
system  characteristics  and  related  quality  factors  are:  If  human  lives 
are  involved  -  dominant  quality  factors  are  reliability  and  correctness: 
if  long  system  life  cycle  is  critical  -  testability,  maintainability  and 
expandability  are  the  dominant  quality  factors. 

3.  Expected  Environment:  Gross  initial  estimate  of  the  development  and 
deployment  environment  (hardware,  software,  communications,  operating 
system11,  etc.)  of  the  target  system. 
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4. 3. 3. 2  CSCI  Analyzer 

This  process  executes  a  Computer  Software  Configuration  Item  (CSCI) 
analysis  based  upon  the  requirements.  The  requirement  decomposition  is 
executed  through  user  interaction  to  a  normalized  level  of  requirements. 
After  decomposition  multiple  recombinations  of  the  sub-requirements  are 
accomplished  to  check  for  testability,  quality  and  project  goal  suitability. 


Inputs  to  the  CSCI  Analyzer  are  described  as  outputs  of  the  Target 

Environment  process  with  the  exception  of: 

1.  Requirements  Feedback:  Detailed  decomposed  requirements  data  as 
available  which  is  used  for  system  level  or  test  level  updates. 

Outputs  from  the  CSCI  Analyzer  are: 

1.  CSCI  Structure:  The  optimum  recomposed  structure  is  identified.  The 
structure  will  be  one  of  a  process  (computer  process,  such  as 
computation,  input / output ,  or  table  look-up),  mission  (such  as  fire 
control,  unit  movement,  logistics,  or  communications)  or  organization 
(such  as  Theater,  Corps,  or  Division  level)  structure  tree. 

2.  Detailed  Requirements:  The  decomposed  expanded  normalized  requirements 
that  form  the  basis  for  program  specifications,  system  requirements  and 
specifications  are  the  basis  for  these  detailed  requirements. 

3.  Decomposed  Requirements:  Low  level  individual  requirements  and  tables 
provided  in  the  T-Tooi  dat3  dictionary  format.  For  example,  a  generic 
requirement  might  be  to  connect  Battlefield  node  A  to  B  using  a  required 
Battlefield  conditions  table,  communications  status  table  and  a  weather 
table.  This  will  be  a  machine  readable  interface,  floppy  disk  or 

ci  mur.un  nations  link. 
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Lines  of  Code:  A  determination  of  the  "rough"  lines  of  code,  either  high 
order  language  or  assembler,  without  comments  compiled.  Accuracy  will  be 
an  order  of  magnitude  rough  estimate. 

Project  Goals:  Realistic  acknowledgment  of  management  goals.  Is  this  a 
rush,  high  risk,  time  sensitive  project?  Is  cost  control  the  key  goal? 

Is  schedule  compliance  the  major  management  goal? 

Quality  Goals / Balance :  The  quality  factors  are  ordered,  expanded,  and 
prioritized.  A  weighting  balance  factor  is  also  applied  based  upon 
system  objectives.  If  correctness  were  a  key  project  quality  factor  -- 
correctness  might  for  example,  be  measured  through  the  design  parameters, 
structured  methodology,  user  friendliness,  and  integration  measures. 
Measurement  methodologies  and  evaluation  procedures  could  then  be 
established  for  each  of  these  evaluation  items.  Therefore,  one  of  the 
design  parameters  might  be  capability  --  as  demonstrated  by  stress 
testing . 

Rough  Schedule:  A  development  time  table  that  depicts  all  the  major 
checkpoints  with  an  identification  of  the  testing  constraints /windows  to 
government  acceptance  testing.  The  schedule  will  be  un-optimized  and  not 
success  biased.  Plan  to  estimate  the  "program  design"  phase  fairly 
accurately,  then  move  forward  in  build  and  install  increments. 

Test  Data  Estimate:  A  first  estimate  of  test  data  requirements  based 
upon  the  requirements  decomposition.  This  breakdown  and  identification 
of  detailed  requirements  (requirements  based  testing)  is  the  basis  for 
the  test  data  estimate.  Specific  input  data  items  include  input 
scenarios  and  operator  input  messages.  The  Test  Data  Collection 
Requirements  captures  the  results  of  these  inputs. 


4. 3. 3. 3  T  Tool 


T  Tool:  The  T  Tool  is  an  off-the-shelf  package  produced  by  Programming 
Environments,  Inc.  (PEI)  and  will  be  integrated  into  this  architecture. 

The  T  Tool  is  a  PC  based  software  tool  that  automatically  designs, 
generates,  traces,  and  documents  software  test  cases  from  a  system 
requirement.  It  has  proven  effective  in  cutting  time  and  costs  from 
development  and  maintenance  schedules  while  improving  quality.  All 
functions  are  covered,  most  probable  error  coverage  includes  samples  from 
all  coverage  areas,  and  very  high  structure  coverage  is  provided  with  the 
T  Tool.  Users  direct  the  T  Tool  through  an  adaptive  interface  that 
varies  from  menu-based  to  command-line  according  to  its  usage.  Prompts, 
memory  aids,  and  help  messages  guide  the  user  at  all  three  easy  steps. 

The  user  enters  software  descriptions  into  several  different 
f ill-in-the-blank  screens:  a  requirement  statement  screen  and  screens 
for  data,  condition,  event,  and  state  definitions.  A  restricted  English 
sentence  structure  is  provided  on  all  screens.  The  user  never  has  to 
worry  about  sentence,  paragraph,  or  document  structure  or  how  to  position 
the  cursor.  The  T  Tool  automatically  checks  all  dictionaries  and 
requirements  for  completeness  and  consistency.  The  T  Tool  automatically 
produces  a  requirements  specification  that  can  be  printed  or  viewed 
on-line  and  included  in  other  documents. 

Inputs  to  the  T  Tool  are  described  as  an  output  of  the  CSCI  Analyzer  process. 

Outputs  from  the  T  Tool  are: 

1.  T  Tool  Requirement  Statements:  This  data  flow  includes  the  system 
requirements  encoded  in  the  T  Tool  data  dictionary  language.  These 
requirements  and  test  cases  are  blind  to  regression  and  stress  testing, 
and  do  not  have  horizontal  or  hierarchical  links.  These  requirement 
Links  must  be  established  by  other  inputs. 

2.  T  Tool  Test  Data  Cases:  The  results  of  the  T  Tool  analysis  are  based  on 
test  case  requirements  statements.  Emphasis  is  on  test  coverage  and 
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test  case  productivity.  The  T  Tool  generated  test  cases  will  exercise  at 
least  once,  every  requirement  for  1002  function  coverage  and  every  most 
probable  error  coverage  for  1002  coverage.  Every  requirement  and  every 
most  probable  error  category  will  be  addressed  at  least  once. 


4. 3. 3. 4  Metric  Model 


Metric  Model:  This  process  uses  historic  defect  profiles  to  characterize 


the  development  environment.  The  purpose  is  to  evaluate  project  goals, 
address  effectiveness  of  testing  methods,  and  evaluate  testing  tools  in  a 
quantitative  measure.  This  system  is  executed  through  choosing  QA  and 
testing  methods  and  tools  that  fit  the  input  characteristics, 
interactively  evaluate  the  system  behavior  and  refine  goals  based  upon 
evaluation  results.  Testing  metrics  included  are  Cyclomatic  Complexity, 


Information  Volume,  Function  Points  and  others. 


Inputs  to  the  Metric  Model  are  described  as  outputs  of  Target  Environment  and 
CSCI  Analyzer.  Two  additional  inputs  are: 


1.  Project  Assets:  An  identification  of  the  software  testing  assets 

available  to  testing.  This  includes  tools,  personnel  skills,  resources, 
time,  hardware  and  system  assets. 


2.  Metrics  Feedback:  A  result  from  the  Test  Results  Analysis  subsystem. 

This  is  an  interpretation  of  the  testing  results  that  is  used  to  tune  the 


software  metric  model  criteria. 


Outputs  from  the  Metric  Model  are: 


AMC  Test  Indicator  Data:  Army  Materiel  Command  (AMC)  input  for  the 
software  progress-development  and  test  management  indicators  of  AMC 
Pamphlets  70-13  and  70-14  will  be  provided.  The  quality  indicators  and 
time/schedule  metrics  and  their  effects  on  testing  will  be  the  key  areas 


of  interest.  These  indicators  will  be  tailored  for  the  software  test 


description. 
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2.  Assets  Available  (Hardware  and  Systems):  A  delineation  of  projected 
computer  assets  that  are  to  be  used  in  the  test  strategy  generation, 
including  testing  mechanization  assets. 

3.  Error/Fault/Failure  Profiles:  Quantification  of  expected  fault  de¬ 
tection,  error  prevention,  error  profile,  and  failure  data.  Methods 
include  functional  testing,  structural  testing,  and  code  reading.  Tools 
include  chief  programmer  team,  document  library,  code  reuse,  and  program 
design  language.  Fault  type  examples  include  control,  data  and  inter¬ 
face.  Error  classes  include  application,  environment  or  clerical  type 
examples . 

4.  Metric  Tuning:  This  is  feedback  on  the  use  of  current  software  testing 
metrics  results.  This  input  addresses  specific  recommendations  as  a 
minimum  for  integration,  acceptance,  and  test  procedure  modification. 
Metric  tuning  also  includes  test  indicator  data  to  indicate  quality 
factors  for  data  base  development,  scheduling  metrics  and  efforts  on  raw 
testing.  These  indicators  will  be  tailored  to  the  software  test  plan. 
Most  probable  error  statistics  data  is  also  generated  by  the  Metric  Model 
and  tailored  to  the  software  design/development  methodology  (i.e.,  00D, 
Top-Down)  and  Language(s)  of  implementation  (i.e.,  Ada,  Fortran, 

Assembler ) . 

5.  Projected  Goals:  Based  upon  the  project  environment  and  monitoring  the 
methods  and  tools  for  testing,  projected  goals  that  affect  the  project 
outcome  are  recommended.  A  question,  metric,  and  goal  paradigm  is  used 
to  interpret  results  and  provide  a  framework  for  iterative  refinement 
through  feedback. 

6.  Tools:  Specific  software  code  or  programs  to  be  used  as  recommended  by 
test  metrics,  resources,  time  and  other  modeled  factors.  The  most 
important  component  is  functional  expertise.  Quality  Assurance  (QA)  must 
have  knowledgeable  "user"  personnel  to  ensure  the  "right"  system  is 
built.  Identification  of  the  best  fit  Quality  Assurance  tools  is 
critical  to  monitor  system  development.  Ensuring  the  relevance  and 
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balance  of  the  QA  reporting  and  compliance  mechanism  is  a  tool 
responsibility. 


4. 3. 3. 5  Test  Development 


Test  Development:  This  subsystem  (process)  is  composed  of  three  major 
components:  test  resources;  test  plans;  and  test  procedures.  These 
components  have  been  the  emphasis  of  this  research.  See  an  expansion  of 
this  subsystem  in  the  Test  Development  Subsystem  detail  chart  attached  at 
Figure  5. 


Inputs  to  Test  Development  are  the  outputs  from  the  T  Tool;  CSCI  Analyzer; 

and  the  Metric  Model  previously  identified.  Additional  inputs  include: 

1.  User  Interaction  -  Test  Design:  This  dataflow  is  a  dialog  between  the 

user  and  the  EXTEND  system.  The  system  architecture  requires  a  minimum 
number  of  data  inputs  to  create  the  resultant  test  plan. 

2.  Detail  Design  Documents:  These  document  will  include  as  a  minimum  the 

following;  Target  System  Operators /User  manual.  Target  System  size  and 
complexity  data,  system  and  program  design  details,  other  test  assets 
available,  integration  strategy  baseline,  and  software  builds  documents. 

3.  Lower  Level  Test  Results:  This  is  the  result  of  previous  level  testing 
(system,  acceptance,  or  user).  This  is  used  within  the  Test  Design 
Subsystem  to  match  with  regression  tests  and  regression  test  data  to  test 
the  integrity  of  lower  level  test  results. 

4.  Test  Analysis  Feedback:  The  feedback  is  a  critical  analysis  of  the  test 
results  and  test  data  collected  from  the  previous  series  of  tests.  The 
dataflow  response  in  a  timely  way  allows  for  tailoring  of  the  test  design 
to  analyzed  problem  areas. 

5.  CM  Status:  From  a  Configuration  Management  (CM)  interface  subsystem  that 
links  with  an  off-the-shelf  CM  tool,  like  Expertware’s  CM  Toolkit.  CM 
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status  provides  the  test  bed  status,  results,  and  system  control 
informtion.  Also  provided  is  problem  reporting,  cross  reference,  version 
description,  and  build  specification  tool  status.  Configuration 
Identification  and  Control  is  maintained  for  soitware,  hardware, 
interfaces,  support  equipment,  instrumentation,  data  bases,  and  all 
diagrams . 


6.  Program  Schedule/Status:  A  user  update  from  an  established  software 
development  model  like  the  Putnam  Cocomo  model,  the  design  element,  or 
others.  A  schedule  for  the  subject  development  effort  is  provided.  This 
input  may  come  from  a  tool  which  may  be  an  off-the-shelf  component 
interfaced  with  the  system.  A  management  input,  separate  from  a  model, 
with  real  subject  project  data  will  also  be  a  component  of  this  input. 

The  program  schedule/ status  also  includes  a  design  update  interactive 
dialog  that  provides  for  the  off-the-shelf  interface  to  system  update 
design  decisions  and  constraints. 

7.  Regression  Test  Update:  The  CM  input  for  regression  testing  including, 
as  a  minimum,  the  test  version  and  test  build,  test  data,  and  thread 
control.  It  includes  previous  test  input,  output,  and  automatic 
verification,  and  may  include  a  key  stroke  saver  function. 

8.  Test  Traceability:  From  the  off-the-shelf  Tools  Interface  subsystem  to 
the  other  Computer  Aided  Software  Engineering  (CASE)  tools  used  in  the 
development.  This  may  range  from  a  spreadsheet  to  a  data  base  to  a 
project  management  or  CM  tool  to' control  the  testing  traceability  and 
control . 

Outputs  from  Test  Development  process  are  discussed  relative  to  the 
subordinate  Test  Devlopment  Subprocesses;  Test  Resources,  Test  Plans  and  Test 
Procedures . 

4. 3. 3. 5.1  Test  Resources 

This  subprocess  accomplishes  the  identification  and  utilization  of  all 
available  testing  assets.  Time  and  schedule  issues  must  be  analyzed  with 
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knowledge  of  test  design  and  system  requirements. 

Inputs  were  discussed  as  inputs  to  the  Test  Development  Process. 

Outputs  from  Test  Resources  are: 

1.  Allocated  Resources:  The  results  of  a  feedback  function  between  test 
resources  and  test  plans  and  procedures  subsystems  that  perform  a  what-if 
analysis  of  the  existing  resources.  Specific  input  items  include 
schedule,  personnel,  software,  instrumentation,  hardware,  test  drivers, 
facilities,  and  interfacing  systems  data. 

2.  Test  Limitations:  Prioritization  of  all  testing  features.  Features  or 
significant  combinations  of  features  that  cannot  be  tested  will  be 
identified.  A  reason  will  be  provided  with  a  probable  risk  assessment. 

3.  Test  Schedule/Budget  Data:  A  critital  path  type  testing  schedule  that 
addresses  time,  resources  and  priorities. 

4. 3. 3. 5. 2  Test  Plans 

This  subprocess  performs  the  quantification  and  mapping  of  all  test 
issues  and  criteria,  test  cases,  test  strategy  and  the  kind  of  test  to 
the  system  requirements.  The  system  requirements  include  both 
operational  and  functional  specifics  such  as  performance,  response,  and 
capacity.  Based  upon  user  interaction,  software  test  descriptions  and 
software  test  procedure  documents  will  be  outputs.  The  Test  Plans 
process  is  broken  into  the  following  sub-processes:  input  correlation, 
requirements  analysis,  test  planning,  knowledge  based  processing,  user 
interaction,  and  test  report  generation.  Components  include  Test 
Strategy  and  Test  Issues  and  Criteria  sub-subprocesses.  Test  Issues  and 
Criteria:  This  process  identifies  the  specific  test  issues,  based  upon 

the  CSCI  structure,  system  requirement  and  system  design  whose  impact 
should  be  addressed  by  the  test  strategy.  The  test  criteria 
determination  is  provided  as  test  case  input  to  test  strategy  and 
integration  planning. 
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Inputs  to  Test  Plans  were  described  as  inputs  to  the  Test  Development  Process 
except : 


1.  Procedure  Feedback:  This  is  an  internal  evaluation  loop  within  the  Test 
Development  subsystem. 

Outputs  from  Test  Plans  are: 

1.  Acceptance  Test  Criteria:  The  specific  user  criteria  to  allow  a 
determination  of  the  system  capability  to  support  the  functional 
requirements.  The  form  and  measurement  of  the  criteria  will  be  defined 
and  quantified.  Risks  will  also  be  defined  and  quantified.  Based  upon 
the  stated  acceptance  test  criteria. 

2.  Integration  Strategy:  This  will  be  a  component  of  the  software  test  plan 
showing  values  covering,  for  example:  integration  testing,  time,  and 
requirement  functionality  testability.  A  project  unique  integration 
strategy  balance  of  testing  must  be  identified.  Integration  Strategy 
Baseline  is  based  on  system  criticality,  software  design  and  testing 
approach.  Critical  components  and  interfaces  must  be  formally  tested. 

The  high  or  low  impact  of  the  software  failing  must  be  acknowledged.  The 
update  is  appropriate  to  development  and  qualification  testing.  The 
result  is  only  a  recommendation,  which  may  require  a  contract 
modification  to  implement  on  the  subject  system  under  development.  This 
dataflow  defines  the  orchestration  of  integration,  system,  and  acceptance 
testing.  Also  identified  is  who  performs  what  functions,  such  as 
approving  the  results,  of  each  test. 

3.  Requirements  Traceability  Matrix:  A  matrix  of  the  detailed  requirements 
to  the  test  strategies.  The  sequence,  level  of  detail,  complexity, 
limitations,  and  other  parameters  will  be  addressed  for  each  detailed 
requirement . 

4.  Regression  Test  Update:  Interdependencies  are  identified  of  the  tests 
designed,  analyzed  and  reported.  Test  cases  from  all  prior  tests  may  be 
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repeated,  and  new  ones  identified.  Regression  testing  validates  new  or 
modified  requirements  that  necessitated  change,  while  ensuring  existing 
requirements  have  not  been  invalidated.  Regression  Testing  is  an 
iterative  process  that  ensures  the  testing  baseline  is  not  corrupted. 


5.  Software  Test  Plan:  This  data  flow  is  the  result  of  the  Test  Plan  and 
Procedures  Subsystem.  The  Test  Plan  follows  the  formt  of  Data  Item 
Description  DI-MCCR-80014 .  However,  the  format  shall  be  flexible  enough 
to  readjust  as  format  changes  are  determined  by  the  user. 

6.  Test  Schedule/Budget:  A  recommendation  from  the  test  strategy  viewpoint 
of  where  testing  emphasis  should  be  placed.  An  acknowledgment  of  the 
impact  on  testing  of  schedule,  resource,  and  technical  risks  attendant  to 
the  specific  ongoing  development.  The  emphasis  is  on  testing  time  and 
the  risk  effect  on  testing  coverage  and  the  telescoping  of  any  test 
slippages.  The  test  schedule  provides  tradeoffs  of  increased  risks 
(short  cuts)  to  the  requirements  and  quality  factors.  As  Fred  Brooks  has 
stated,  "More  software  projects  have  gone  awry  for  lack  of  calendar  time 
than  for  all  other  causes  combined." 

7.  Test  Strategy  Report:  Recommendation  of  the  testing  mix,  with 
percentages  for  the  current  testing.  This  includes  code  walkthroughs, 
code  reading,  document  reviews,  black  box  and  glass/white  box  testing, 
performance  testing,  etc.  The  test  approach  will  be  delineated  for  unit, 
system,  integration  and  acceptance  testing.  Software  build  testing 
status  and  relationship  with  all  key  development  milestones  will  be 
addressed . 


8.  Test  Planning  and  Preparation:  All  the  information  required,  in  a 

"generic"  mode,  for  test  running  on  the  subject  system  testbed.  This 
includes  the  specific  test  parameters,  environment,  procedures  and  data 
for  a  specific  test. 
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9.  Test  Suspension/Resumption:  This  dataflow  identifies  the  criteria  used 
to  suspend  all  or  a  portion  of  the  testing  activity.  The  resumption 
element  specifies  the  testing  activities  that  must  be  repeated  when 
testing  is  resumed  --  and  any  preconditions  for  resumption. 

4. 3. 3. 5. 3  Test  Procedures 

The  test  procedure  development  that  describes  all  inputs,  outputs  for  the 
System  testing  and  produces  the  Software  Test  Procedures.  Inputs  were 
described  as  inputs  to  the  Test  Development  Process.  The  Output  from 
Test  Procedures  is: 

1.  Software  Test  Procedures:  The  test  procedures  that  are  executed  along 
with  the  test  data,  test  time,  and  other  input  to  be  run  on  the  target 
system  or  testbed  system. 

4. 3. 3. 6  Testbed/Interface 

Testbed/Interface:  This  process  executes  the  preparation  and  formatting 
of  all  the  data  required  to  allow  the  physical  running  of  tests  on  the 
subject  development  system  testbed  or  the  target  system  if  available. 

This  process  result  may  be  provided  in  a  machine  readable  format  or 
through  a  telecommunication/network  type  link. 

The  input  to  the  Testbed/Interface  Process  was  described  as  an  output  of  the 
Test  Development  process  the  Test  Planning  and  Preparation  data  flow. 

Outputs  from  Testbed  Interface  are: 

1.  Test  Data  Collection:  The  total  data  collection  from  the  subject  test. 
This  includes  the  original  test  planning  and  preparation  input  from  the 
test  design  subsystem. 

2.  Test  Results:  The  total  results  of  the  subject  test,  in  a  machine 
readable  format. 
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4. 3. 3. 7  Off-The-Shelf  Tools  Interface 


Off-The-Shelf  Tools  Interface:  This  interface  process  allows  for  the 
interaction  of  the  EXTEND  Testing  system  and  currently  developed  standard 
product  offerings  of  Configuration  Management,  Computer  Aided  Software 
Engineering,  and  schedule/resource  management  tools. 

The  input  to  Off-The-Shelf  Tools  Interface,  the  regression  test  update 
dataflow,  was  previously  described  as  an  input  to  the  Test  Development 
process . 

Outputs  from  Off-The-Shelf  Tools  Interface  have  been  previously  identified  as 
inputs  to  the  Test  Development  process  and  also  include: 

1.  Regression  Test  Requirement:  The  CM  input  including  the  test  version  and 
test  build,  test  data  and  thread  control  information. 

4. 3. 3- 8  Test  Results  Analysis 

Test  Results  Analysis:  This  subsystem  (process)  compares  the  test 
procedure  expected  test  result  with  the  actual  test  result.  It  verifies  that 
regression  test  requirements  were  included.  The  results  of  this  analysis 
provide  the  feedback  to  the  other  subsystem  to  tune  the  testing  effort. 

Inputs  to  the  Test  Results  Analysis  Process  are  described  as  outputs  of  the 
Off-The-Shelf  Tools  Interface,  Testbed/Interface,  Metric  Model,  and  Test 
Development  Processes. 

Outputs  from  Test  Results  Analysis  are: 

1.  On/Off  Schedule  Determination:  A  simple  determination  of  testing 

proceeding  on  or  off  the  testing  schedule.  The  determination  will  be 
explained  with  specific  problem  areas. 
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Software  Quality  Determination:  An  analysis  of  the  current  development 
ability  to  test  and  verify  the  dominant  quality  factors. 

Improvement  Actions:  The  actions  available,  with  resource  and  time 
components,  to  improve  the  testing  effort. 

Regression  Analysis:  A  determination  that  the  subject  system  development 
integrity  has  been  validated  and  that  the  development  process  has  not 
invalidated  the  baselined  system  components. 
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4.4  Prototype  Software  Testing  Expert  System 
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The  following  section  is  a  sample  transcript  from  the  prototype  expert 
system  shell.  The  purpose  is  to  show  the  highly  user  interactive  nature 
available  and  the  level  of  user  assistance  resident  in  the  shell.  A 
rephrased  question,  summary  of  the  current  consultation  (dialog),  partial 
conclusion’s  list  from  the  expert  system,  and  the  ability  to  back-up  to  the 
previous  question  are  ail  options  available  to  the  user. 

An  expert  system  shell  was  used  to  assist  in  prototyping  the  interactive 
session  of  the  EXTEND  system  for  user  testing  data  input.  The  shell  is 
Sonex  software  written  in  the  MULISP  (Soft  Warehouse,  Inc.)  implementation 
of  LISP.  The  shell  models  a  decision  tree  structure.  The  shell  uses  a 
modified  "If  A  Then  B  Else  C"  rule  form. 

Results .  This  initial  prototype  software  testing  expert  system  effort  had  no 
architecture  and  no  direction.  The  prototype  development  had  nothing 
substantial  to  relate  itself  with  until  a  testing  architecture  had  been 
defined.  Therefore,  the  decision  was  made  to  halt  this  research  thrust  and 
continue  with  expanding  the  overall  architecture  displayed  in  Section  4.3  of 
this  report. 
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*  *  *  SOFTWARE  TESTING  ASSISTANT  SYSTEM  *  *  * 

To  evaluate  the  software  project  characteristics,  design  methodology  and 
software  reuse  factors  to  access  most  probable  error  statistics  and  define 
acceptance  test  criteria. 

In  using  the  program  you  will  be  asked  to  answer  guestions  about  the  existence 
of  various  kinds  of  evidence.  Answers  to  questions  can  be  “Y",  "N",  or  if  you 

dont  know,  "D".  To  quit,  enter  "Q“ .  In  addition  to  supplying  answers,  you  can 
request  information  at  any  time  with  the  following  commands: 

?  --  Print  a  rephrased  version  of  the  question 
A  --  Access  database  to  assist  in  answering  the  question 
S  --  Print  a  summary  at  this  point  in  the  consultation 
T  --  Let's  you  see  EXTEND' s  partial  conclusions 
U  --  Stops  the  trace 

W  --  Show  answer  to  a  previous  question 
B  --  Break  the  consultation  and  put  you  in  MuLISP 

Press  RETURN  to  continue  . . . 

EXTEND  SOFTWARE  TEST  PLAN  GENERATOR 

project  data  options 

Number  inputs 

1  PROJECT-CHARACTERISTICS 

2  TEST-PARAMETER-DESIGN 

3  SOFTWARE-REUSE-FACTORS 


Enter  number  or  Q  to  quit: 

2 

The  following  is  intended  to  recommend  test  parameter  design  criteria  tailored 
on  your  situation 

1  --  For  which  of  the  following  do  you  have  any  information: 

1)  Size  of  the  development 

2)  Design  Methodology 

3 )  Other  Factors 

Please  enter  one  or  more  of  the  preceding  numbers 

(separated  bv  blanks  and  terminating  with  a  <CR>  1 

2 

3 


Please  think  in  terms  of  defining  development  size  as  large,  medium  or  small. 

2  --  Is  this  project  a  small  development?  (Y/N/D/A/S/T/U/W/Q/B/?)  : 

N 

3  —  Is  this  a  medium  sized  development?  (Y/N/D/A/S/T/U/W/Q/B/?)  : 

Y 

4  --  Software  development  type  is  a  Command- 
Control-Communications-Intelligence  C3I  system?  (Y/N/D/A/S/T/U/W/Q/B/?)  : 

N 

5  --  Software  development  type  is  a  Communications  type  system? 

(Y/N/D/A/S/T/U/W/Q/B/?)  : 

N 

6  --  Software  development  type  is  a  Field  Artillery  type  system? 

(Y/N/D/A/S/T/U/W/Q/B/?)  : 

N 

7  --  Software  development  is  an  Intelligence/  Signal  type  system? 

(Y/N/D/A/S/T/U/W/Q/B/?)  : 

Y 

8  --  For  which  of  the  following  do  you  have  any  information: 

1)  Object  Oriented  Design 

2)  Yourdon-DeMarco  Design 

3)  Top  Down  Design 

4)  Middle  Out,  Iterative 

5)  Combination  Design 

Please  enter  one  or  more  of  the  preceding  numbers 

(separated  by  blanks  and  terminating  with  a  <CR>  3 


9  --  Are  there  other  factors,  besides  size  &  design  Methodology,  that  effect 
this  development?  (Y/N/D/A/S/T/U/W/Q/B/?)  : 


Based  on  your  responses,  my  evaluation  is  as  follows: 


1)  Very  exacting  controls  required  on  SIGINT  development 

2)  Standard  design,  testing  better  not  be  top-down.  If  testing  is  also 
top-down  then  give  special  IW  and  Govt,  personnel  attention. 

3)  Evidence  of  project  to  include  the  requirements  of  AMC-P  70  -13  and  70 


You  have  established: 

4)  Size  of  the  development 

5)  No  small  development  size 

6)  medium  system  development 

7)  No  c3l  development 

8)  No  comm  system  development 

9)  No  field  Artillery  system  development 

10)  SIGINT  system  development 

11)  Design  Methodology 

12)  Top  Down  Design 

13)  Other  Factors 

14)  Identify  other  factors 


Do  you  wish  to  consider  another  inputs  or  another  topic  (Y/N/Q/B/?): 
N 
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4.5  Estimate  of  Architecture  Feasibility 

The  subject  of  software  testing  has  not  been  an  area  of  major 
architecture  research  of  expert  systems  application.  There  are  multiple 
probable  reasons  why  this  is  true.  The  principle  reason  is  that  software 
testing  is  not  generally  considered  an  end  upon  itself.  The  typical  software 
developer  looks  at  software  testing  as  a  necessary  evil,  not  an  integral  part 
of  the  system  development  process.  We  believe  this  mentality  that  testing  is 
only  a  "drain  on  development  resources"  is  very  pervasive.  This  negative 
climate  has  created  the  vacuum  of  significant  architecture  expert  systems 
work  in  the  software  testing  domain. 

We  believe  the  EXTEND  System  is  feasible,  because  the  system  components 
are  feasible.  Refering  back  to  Figure  6,  the  EXTEND  Execution  Flow,  the  key 
components  of  the  system  are  the  separate  subsystems,  the  user  interface,  and 
the  definition  of  the  testing  threads.  The  EXTEND  system  consists  of  the 
Target  Environment,  CSCI  Analyzer,  Metric  Model,  T  Tool,  the  Test  Development 
Testbed/Interface,  Off-the-shelf  Tools  interface,  and  Test  Results  Analysis 
Subsystems . 

The  Target  Environment  is  a  modest  effort  that  begins  the  target  system 
definition,  and  provides  default  values /conditions  and  assumptions  when  the 
user  can't  provide  specific  data.  The  CSCI  analyzer  defines  the  Target 
System  software  development  structure  options  and  has  commonality  with  the 
DIOGENES  system  discussed  in  the  Section  2.0  Other  Current 

Research/Developments.  Our  implementation  will  build  from  DIOGENES  lessons 
learned  and  may  integrate  a  portion  of  the  DIOGENES  system  knowledge 
base/logic  structure.  The  Metric  Model  subsystem  is  based  upon  current  work 
ongoing  at  the  University  of  Maryland,  the  TAME  (Tailoring  and  Ada 
Measurement  Environment)  system.  The  EXTEND  system  development  will  use 
germaine  knowledge  base  logic  and  structure  from  the  TAME  system. 

The  T  Tool  is  produced  by  PEI.  The  EXTEND  development  will  directly 
interface  and  include,  the  T  Tool. 

The  Test  Development  Subsystem  would  provide  the  EXTEND  System  final 
output  of  the  software  test  description.  This  component  has  no  direct 
precident  that  we  are  currently  aware  of.  The  Sonex  approach  was  to  focus 


4  3 


~\  * 


« 


on  the  Test  Development  subsystem  for  the  Phase  X  SBIR  research.  This 
continued  decomposition  of  the  test  development  subsystem  has  produced  a 
better  understanding  of  the  total  EXTEND  knowledge  requirements. 

The  generic  feasiblity  input-process-output  diagram  is  depicted  in 
Figure  7.  The  outputs  identified  in  Figure  7  are  now  discussed  in  terms  of 
the  EXTEND  system  architecture. 

Identification  of  most  probable  error  statistics:  This  will  be 
accomplished  within  the  Metric  Models  subsystem  of  the  EXTEND  architecture. 
This  data  will  be  based  upon  the  research  provided  in  the  University  of 
Maryland  TAME  project.  This  measurement  and  evaluation  environment  will 
provide  probable  error  statistics.  Based  upon  user  input,  target 
environment,  and  CSCI  analysis  user  inputs  or  default  values,  the  conditions 
will  be  generated.  These  values  will  be  passed  to  the  T  Tool  and  Test 
Development  Subsystem,  and  the  default  conditions  and  assumptions  provided  in 
the  software  test  description  appendix  of  assumptions  and  constraints.  This 
feature  will  be  implemented  in  the  Phase  II  SBIR  development. 

Definition  of  Acceptance  Test  Criteria:  The  design  of  the  acceptance 
testing  at  the  CSCI  level  shall  be  the  major  focus  of  the  EXTEND  System  Phase 
II  development.  The  software  test  description  that  is  generated,  the  experts 
contacted  and  the  knowledge  extracted  will  focus  on  the  CSCI  Acceptance  level 
testing. 

Integration  Plan:  This  will  be  partially  implemented  through  the 
resultant  software  test  description  and  constraints.  The  effect  of  different 
user  inputs  and  the  resultant  generation  of  default  values  will  provide  an 
ability  to  compare  integration  plan  strategy  results  as  depicted  within  the 
software  test  description,  and  the  captured  experience  of  experts. 

Identification  of  multiple  Test  Strategies:  This  will  be  partially 
implemented  in  the  Phase  II  SBIR  EXTEND  development  through  the  software  test 
description  product.  The  different  user  inputs,  and  the  built-in  default 
conditions  and  strategies  will  be  executed.  The  criteria  and  conditions  that 
identified  a  specific  strategy  will  be  captured  in  the  software  test 
description  appendix  of  assumptions  and  constraints. 
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5.0  CONCLUSIONS 


The  conclusions  fall  into  two  categories,  conclusions  on  testing  and 
conclusions  on  knowledge  based  system  development. 

Testing .  The  problem  definition  stage  of  this  research  was  the  hardest  task. 
We  believe  the  testing  field  is  very  splintered,  but  generally  accepted 
engineering  principles  do  exist.  The  practitioners  in  the  testing  domain  are 
artisans  who  do  what  they  do  because  it  has  worked  before.  These  testers 
find  it  hard  to  explain  why.  The  testing  domain  uses  reasoning  by  analogy  to 
a  great  extent.  There  seems  to  be  global  commonality  of  overall  testing 
approaches.  Certain  truths  are  acknowledged  as  discussed  in  Section  4.1.6 
Results  of  literature  reviews  and  state  of  the  art  survey.  However,  an 
architecture  to  tie  the  testing  and  control  functions  together  in  the  soft¬ 
ware  development  process,  is  missing. 

Knowledge  Based  Systems.  Artificial  Intelligence  and  Expert  Systems  have 
been  the  focus  of  much  recent  activity.  There  is  value  in  the  knowledge 
based  system  development  --  but  it  is  a  difficult  process  to  implement. 

There  are  two  major  hurdles  in  knowledge  based  system  development:  The  scope 
definition  and  the  expert  knowledge  extraction  problems.  We  have  addressed 
the  execution  of  testing  from  a  global  point-of -view.  Our  approach  for 
addressing  the  knowledge  based  development  is  as  follows: 

1.  We  have  identified  the  key  component  of  testing,  and  labeled  it  the  Test 
Development  process  in  the  EXTEND  architecture. 

2.  Address  first  and  further  define  a  well  understood  smaller  segment  of  the 
Test  Development  process,  the  Test  Plan  subprocess. 

3.  Define  the  environment  for  the  Test  Plan  subprocess. 

4.  Structure  the  environment  through  the  use  of  logic  trees,  Data  Flow 
Diagrams,  and  other  techniques. 
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5.  Identify  one  very  senior  testing  expert  to  critique  the  representation  of 
the  Test  Plan  subprocess. 


6.  Icerate  the  above  approach  until  all  the  subprocess  facets  have  been 


addressed. 


7.  Continue  to  refine  the  detail  in  the  logic  trees  and  other 

representations  with  detailed  domain  facts  from  testing  experts. 


8.  Identify  the  voids  and  inconsistencies  in  the  compiled  knowledge. 


9.  Identify  the  "best"  or  most  knowledgeable  expert  to  fill  the  identified 
voids  in  the  testing  knowledge. 


10.  Execute  all  the  above  activities  in  parallel  with  user  interface  proto¬ 


typing. 


6 . 0  RECOMMENDATIONS : 


The  findings  of  the  Phase  I  effort  justify  the  recommendation  that  the 
EXTEND  system  be  implemented  as  a  Phase  II  of  the  current  SBIR  program. 

The  recommended  approach  includes: 


Implementation  of  a  full  scale  knowledge  engineering  effort  to 
develop  a  prototype  based  on  the  Test  Plan  subsystem. 

Focus  of  the  implementation  on  the  key  testing  issues,  for  example, 
metrics,  test  strategies,  and  automation  assistance. 

Interface  with  the  T  Tool  and  other  CASE  tools  to  capitalize  on 
off-the-shelf  and  ongoing  development  efforts. 
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of  correctness  of  the  resulting  code.  The  refinement 
paradigm  requires  Intermediate  level  constructs  and 
search  for  efficient  Implementations,  which  are  discussed. 
An  example  Is  devised  to  see  If  macropara  11  el  I sm  In  the 
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1461  WEGNER,  PETER , EDITOR ;  "RESEARCH  DIRECTIONS  IN  SOFTWARE 
TECHNOLOGY  -  INTRODUCTION  AND  OVERVIEW,"  In  3RD  INT'L 
CONFERENCE  ON  SOFTWARE  ENGINEERING.  0(0):  May  1978.  PP. 
1-38.  Also  in  RESEARCH  DIRECTIONS  IN  SOFTWARE  TECHNOL¬ 
OGY.  0(0):  Jan  1979.  Avail,  from  MIT  Press,  28  Carleton 
Street,  Cambridge,  MA  02142. 

Keywords:  DATABASE  MANAGEMENT  SYSTEMS,  ARCHITECTURE, 
AUTOMATIC  PROGRAMMING,  PERFORMANCE,  PROGRAM  SYNTHESIS, 
DEVELOPMENTAL  METHODOLOGIES,  NATURAL  LANGUAGES,  DISTRI¬ 
BUTED  PROCESSING,  CONCURRENT  PROGRAMMING,  TESTING, 
SPECIFICATIONS,  MANAGEMENT,  SOFTWARE  ENGINEERING,  COM¬ 
PLEXITY,  EDUCATION 
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The  organization  of  this  book  is  explained,  the  changing 
technological  environment  is  reviewed,  and  capsule 
descriptions  of  each  paper  (chapter)  are  included  in  this 
introduction.  The  first  four  chapters  (Part  I)  consider 
the  nature  of  the  software  problem  and  describe  concepts 
and  tools  for  managing  large  software  systems.  The 
remaining  16  chapters  (Part  II)  describe  and  analyze 
specific  research  areas,  divided  into  three  subareas:  (1) 
software  methodology  (managerial  and  technical  issues); 

(2)  computer  system  methodology  (computers,  languages, 
system  programs);  and  (3)  application  methodology  (broad 
application  areas).  Discussion  items  are  included  at  the 
end  of  each  Part  or  subarea  which  further  exolcre 
specific  research  areas  or  offer  alternative  points  of 
view. 

1784  HOFFMAN,  KARLA;  WITZGALL ,  CHRISTOPH;  "A  LEXICAL  SYNTHESIS 
APPROACH  TO  USER-ORIENTED  INPUT  SPECIFICATION , "  In  17TH 
ANNUAL  TECHNICAL  SYMPOSIUM ,NBS/ ACM.  0(0):  Jun  1978.  PP. 
179-185.  Avail,  from  ACM,  Inc.,  1133  Avenue  OF  Americas, 
New  York,  NY  10036. 

Keywords:  TEXT-PROCESSING  APPLICATIONS,  CODE  READING, 

CODE  VERIFICATION,  NATURAL  LANGUAGES 

This  paper  presents  a  general  and  highly  flexible  "lexi¬ 
cal  synthesis"  approach  to  the  lexical  decoding  problem 
based  on  systematic  string  recognition  rather  than  delim¬ 
iting  rules.  It  has  successfully  been  implemented  in  an 
operating  general-purpose  lexical  synthesis  package  ULEX. 

3939  CHAPMAN,  DAVID;  "A  PROGRAM  TESTING  ASSISTANT,"  COMMUNICA¬ 
TIONS  OF  THE  ACM.  25(9):  Sep  1982.  PP.  625-634. 
Contract/Grant  No.  MCS-7912179,  Sponsored  by  NATIONAL 
SCIENCE  FOUNDATION.  Contract /Grant  No.  N00014-B0-C-0505  , 
Sponsored  by  Office  of  Naval  Research,  Quincy  Street, 
Arlington,  VA  22217. 

Keywords:  TRANSFORMATION,  LISP,  SOFTWARE  TOOLS,  PROGRAM¬ 
MING  AIDS,  DEBUGGING,  SOFTWARE  ENGINEERING  TOOLS  AND 
TECHNIQUES,  ARTIFICIAL  INTELLIGENCE,  AUTOMATED  TESTING 

This  article  describes  the  design  and  implementation  of  a 
program  testing  assistant  for  the  MIT  Artificial  Intelli¬ 
gence  Laboratory's  LISP  Machine.  The  program  testing 
assistant  aids  a  programmer  in  the  definition,  execution, 
and  modification  of  test  cases  during  incremental  program 
development.  A  brief  overview  of  the  testing  assistant 
is  first  presented.  Next,  an  example  scenario  of  the 
assistant's  use  is  given,  with  commentary  and  additional 
explanation  of  topics  introduced  in  the  overview,  and 
implementation  issues  and  techniques  are  examined. 
Finally,  the  testing  assistant  is  related  to  other 
research. 
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4615  DEAN,  JEFFREY  S.;  MCCUNE,  BRIAN  P.;  ADVANCED  TOOLS  FOR 
SOFTWARE  MAINTENANCE.  Technical  Report  TR-3006-1. 
Contract /Grant  No.  F30602-80-C-0176 ,  Sponsored  by  Rome 
Air  Development  Center,  Griff iss  AFB,  Rome,  NY  13441- 
5700.  Avail,  from  National  Technical  Information  Service 
5285  Port  Royal  Rd ,  Springfield,  VA  22161. 

Keywords:  UNIX,  PRODUCTIVITY,  VERIFICATION,  EDITORS,  PRO¬ 
GRAM  MAINTENANCE,  MODIFICATION  PROCEDURES, 

COMMAND, CONTROL,  AND  COMMUNICATIONS,  TESTING,  AUTOMATED 
DOCUMENTATION,  KNOWLEDGE  BASED  SYSTEMS,  ARTIFICIAL  INTEL¬ 
LIGENCE,  INTERLISP.  MAINTENANCE  TOOLS  AND  TECHNIQUES,  ADA 

This  report  discusses  software  maintenance  and  proposes 
maintenance  tools  and  techniques  for  the  Ada*  programming 
environment.  Maintenance  practices  for  several  Air  Force 
Command,  Control,  Communications,  and  Intelligence  (C3I) 
software  projects  are  reviewed.  Three  out  of  the  four 
major  problems  identified  during  the  project  were  attri¬ 
buted  to  the  difficulty  of  comprehending  software.  Nine 
tools  are  proposed  to  help  solve  these  and  other  prob¬ 
lems,  including  a  tool  to  help  coordinate  the  programming 
process,  a  tool  to  aid  in  the  collection  and  use  of  docu¬ 
mentation,  and  an  editor  that  is  knowledgeable  about  what 
it  is  editing.  The  tools  are  based  on  related  technolo¬ 
gies  that  are  also  discussed  in  the  report:  artificial 
intelligence,  automatic  programming,  intelligent  use 
interfaces,  formal  verification,  programming  environ¬ 
ments,  and  software  metrics.  Recommendations  are  made  as 
to  how  the  tools  may  be  incorporated  into  the  Ada  Pro¬ 
gramming  Support  Environment  (APSE).  (  *Ada  is  a  trade¬ 
mark  of  the  U.S.  Department  of  Defense). 

6176  VAN  DERLINDEN,  PETER;  "WRITING  DIAGNOSTIC  SOFTWARE  IN 
ADA, "  ACM  ADA  LETTERS .  4(2):  Sep  1984 .  PP.  44-53 . 

Keywords :  TESTING,  VERIFICATION  TOOLS  AND  TECHNIQUES, 
EXPERT  SYSTEMS,  ARCHITECTURE,  ADA 
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This  paper  desribes  the  VERIFY  432  package,  which  was 
written  in  Ada*,  and  which  evaluates  the  hardware  status 
of  an  HIS  432  board  system  or  an  integrated  MULTIBOX  com¬ 
puter.  Some  observations  are  made  on  the  possibility  of 
building  a  knowledge  base  into  this  software,  to  upgrade 
to  an  expert  system.  A  hardware  diagnostic  package  would 
usually  be  written  in  a  low-level  language,  perhaps  even 
utilising  special-purpose  microcode  functions.  However, 
the  authors'  experience  demonstrated  that  Ada  is  suitable 
for  implementing  this  kind  of  testing  suite,  and  has  many 
features  which  especially  facilitate  more  general  systems 
programming.  Some  of  the  particular  advantages  of  using 
Ada  are  pointed  out,  as  well  as  areas  in  which  the 
language  could  have  provided  more  assistance  than  it  did. 
(author)  ( *Ada  is  a  trademark  of  the  U.S.  Department  of 
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6907  KEIRSEY,  D. ;  MITCHELL,  J.;  BULLOCK,  B .  ;  ‘NUSSMEIER  ,  T.  ; 

TSENG,  D. ;  AUTONOMOUS  VEHICLE  CONTROL  USING  ARRTIFICIAL 
INTELLIGENCE  TECHNIQUES .  Avail,  from  National  Technical 
Information  Service  5285  Port  Roval  Rd ,  Sprincrf  ield ,  VA 
22161.  Mo.  AD-P003  033. 

Keywords:  SYSTEM  TESTING,  ROBOTICS,  ARTIFICIAL  INTELLI¬ 
GENCE 

A  review  of  early  work  or.  a  project  to  develop  autonomous 
vehicle  control  technology  is  presented.  The’  primary 
veal  of  this  effort  is  the  development  of  a  generic  capa¬ 
bility  that  can  be  specialized  to  a  wide  range  of  DoD 
applications .  The  emphasis  in  this  project  is  develop¬ 
ment  of  the  fundamental  Artificial  Intelligence  based 
technology  required  by  autonomous  systems  and  the  imole- 
mentation  of  a  testbed  environment  to  evaluate  and  demon¬ 
strate  the  system  capabilities.  (author! 


